Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: lookup manifest file size #2590

Merged
merged 11 commits into from
Oct 23, 2023

Conversation

QuenKar
Copy link
Contributor

@QuenKar QuenKar commented Oct 12, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

  • use HashMap<String, u64> to keep track for every manifest file size in ManifestObjectStore .
  • we can use RegionManifestManager::manifest_size() to get manifest files size.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

#2551

@codecov
Copy link

codecov bot commented Oct 12, 2023

Codecov Report

Merging #2590 (6a8067b) into develop (6e87ac0) will decrease coverage by 0.48%.
Report is 11 commits behind head on develop.
The diff coverage is 98.56%.

❗ Current head 6a8067b differs from pull request most recent head 8340484. Consider uploading reports for the commit 8340484 to get more accurate results

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2590      +/-   ##
===========================================
- Coverage    85.33%   84.86%   -0.48%     
===========================================
  Files          737      738       +1     
  Lines       118030   119148    +1118     
===========================================
+ Hits        100724   101109     +385     
- Misses       17306    18039     +733     

@QuenKar QuenKar marked this pull request as draft October 13, 2023 03:01
@QuenKar QuenKar marked this pull request as ready for review October 13, 2023 06:22
@QuenKar QuenKar marked this pull request as draft October 16, 2023 02:04
@QuenKar QuenKar force-pushed the feat/manifest-file-size branch from fbf9197 to bd8989f Compare October 16, 2023 02:04
@QuenKar QuenKar marked this pull request as ready for review October 16, 2023 05:07
@QuenKar
Copy link
Contributor Author

QuenKar commented Oct 16, 2023

After discussing with @evenyag , we decide to only calculate manifest size from the latest checkpoint, the files'version <= latest checkpoint will not be included.
There are two main reasons.

  • manifest file size is usually not large
  • there is no need to read the manifest file which version <= latest checkpoint

@QuenKar QuenKar marked this pull request as draft October 16, 2023 06:41
@QuenKar QuenKar marked this pull request as ready for review October 16, 2023 07:58
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
@QuenKar QuenKar force-pushed the feat/manifest-file-size branch from e2068ef to 33ae485 Compare October 17, 2023 02:30
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
src/mito2/src/manifest/storage.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@v0y4g3r v0y4g3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@v0y4g3r v0y4g3r added this pull request to the merge queue Oct 23, 2023
Merged via the queue into GreptimeTeam:develop with commit fbc8f56 Oct 23, 2023
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants